Public: Technology Reviews : Replicating CC subversion repositories to Git
This page last changed on Mar 23, 2009 by stepheneb.

These subversion repositories at CC are automatically synched to public git clones on http://github.com and http://gitorious.org/.

svn repo git mirror public git clone url git-svn clone archives* size of archive (MB)
otrunk-examples otrunk-examples git://github.com/stepheneb/otrunk-examples.git    
DIY DIY git://github.com/stepheneb/diy.git diy.svn.git.tgz 11
ccsailportal ccsailportal git://github.com/stepheneb/ccsailportal.git ccsailportal.svn.git.tgz 5
SDS SDS git://github.com/stepheneb/ccsailportal.git sds.git.tgz 6
CCProjects CCProjects git://gitorious.org/otrunk-examples/mainline.git concord-projects-common.svn.git.tgz 394

There is no direct synching back to svn from the git mirrors.

If you are a developer with commit access to the svn repos you would be better served by replicating parts of this process to make your own local git clone of the svn repository (see the Ruby script process-commit.rb). You can then use the command git svn dcommit to push commits back to the main svn repo.

* Archives (tarred and gzipped) are created once a day (3:12 AM) of the git-svn clones of these repositories. If the svn repository is large and you want to work directly with a git-svn clone (appropriate if you will be making direct commits to the svn repository from your git clone) you can save time by downloading and expanding an archive of the git-svn clone instead of performing the initial import yourself.

This is a brief description of how this works:

Setup

  1. create a git clone of the svn repo locally (this can take a long time)
  2. compress this repo when done: git gc
  3. create an empty git repo on an external hosting service
  4. add a remote reference to the new external repo using an git@ type url (ssh)
  5. git push to the external repo

In order to push to the remote public git repositories I have created a new public/private key and have uploaded a copy of thepublic key to github and gitorious.

Update

  1. cd to project dir
  2. fetch updates to svn repo: git svn fetch
  3. make sure we are on the master branch: git checkout master
  4. rebase just in case someone was doing some hacking: git svn rebase (this shouldn't be necessary)
  5. push the svn changes to git repo: git push remote master

Any external hosting service will require a copy the public key for the user sbannasch on this server to allow pushes via ssh.

The process below is complex because of the work I'm doing to make sure the script pushing commits process-commit.rb is authorized to do so.

I'll bet the curl command in the svn post-commit hook and the php command could be combined into one command. There may also be a better way to start executing the process-commit.rb under an authorized username other than via mail and procmail.

Details

When a svn commit is made the post-commit hook for that repo is called and the following curl command at the end of the file is executed:

file: /home/subversion/<repo-name>/hooks/post-commit

/usr/bin/curl -d "revision=$REV&repository=$REPOS" http://svn.concord.org/cgi-script/post-commit.php

The php script forwards the commit info to my local email account on the same server:

file: /web/svn.concord.org/cgi-script/post-commit.php

<?php
if(($_SERVER['REQUEST_METHOD']==='POST') && (isset($_POST['revision'])) && (isset($_POST['repository']))) {
  $revision = $_POST['revision'];
  $repository = $_POST['repository'];

  $to = "sbannasch";
  $subject = "revision: {$revision} repository: {$repository}";
  $body = "revision: {$revision}\nrepository: {$repository}";

  if (mail($to, $subject, $body)) {
    print "ok ...";
  } else {
    print "mail not sent ...";
  }
} else die("no ...");
?>

The email account sbannasch is not an account I use for general purpose email and in the home directory for the sbannasch user is this procmail script that passes the commit info to a ruby script, process_commit.rb, running under my username.

useful procmail reference: Procmail QuickStart

file: /home/sbannasch/.procmailrc

PATH=/usr/bin:/opt/local/bin:/usr/kerberos/bin:/usr/local/bin:/bin:/bin:/home/sbannasch/bin:\
/usr/java/jdk1.5.0_06/bin:/usr/bin:/usr/local/bin:/usr/sbin:/usr/local/sbin:/sbin
MAILDIR=/var/mail
DEFAULT=/var/mail/sbannasch
SHELL=/bin/bash
LOGFILE=/home/sbannasch/Procmail/pmlog
LOG="My PATH is currently $PATH
My SHELL is currently $SHELL
"
:0
* ^Subject.*revision.*repository
|/usr/bin/env ruby /home/sbannasch/process_commit.rb

The Ruby script process-commit.rb parses the commit info using the mailparser library and executes the local commands to complete the integration.

file: /home/sbannasch/process_commit.rb

#!/usr/bin/env ruby

require 'rubygems'
require 'mailparser'
require 'yaml'

#
# This program uses the Ruby library mailparser:
#
#   http://tmtm.org/downloads/ruby/mailparser/
#
# Documentation: http://www.tmtm.org/ruby/mailparser/
#
# The documentation is in Japanese but the code examples
# are in English.
#
# After downloading and extracting the tar.gz archive
# the program is installed into the Ruby site library 
# using the following commands:
#
#   make
#   sudo make install
#
# There is no native code compilation.
#
LOGGING=true

def chdir(dir, &blk)
  if LOGGING
    File.open("/home/sbannasch/process_commit_log", 'a+') {|f| f.write("cd #{dir}" + "\n")}
  end
  Dir.chdir(dir, &blk)
  if LOGGING
    File.open("/home/sbannasch/process_commit_log", 'a+') {|f| f.write("cd .." + "\n")}
  end
end

def do_command(command)
  if LOGGING
    File.open("/home/sbannasch/process_commit_log", 'a+') {|f| f.write(command + "\n")}
  end
  `#{command}`
end

def update_ccsailportal
  chdir("/home/sbannasch/src/ccsailportal.git") do
    do_command("git svn fetch")
    do_command("git checkout master")
    do_command("git svn rebase")
    do_command("git push github master")
    do_command("git checkout itemservice")
    do_command("git svn rebase")
    do_command("git push github itemservice")
  end
  # chdir("/home/sbannasch/src") do
  #   do_command("nice -n 15 tar czf /web/rails.dev.concord.org/archive/ccsailportal.git.tgz ccsailportal.git")
  # end
end

# How otrunk-examples was setup on otto:
#
#   cd ~/src
#   git svn clone -Totrunk-examples --prefix=svn/ https://svn.concord.org/svn/projects/trunk/common/java/otrunk otrunk-examples.git
#   cd otrunk-examples.git
#   git gc
#   git remote add github git@github.com:stepheneb/otrunk-examples.git
#   git push github master
#
def update_otrunk_examples
  chdir("/home/sbannasch/src/otrunk-examples.git") do
    do_command("git svn fetch")
    do_command("git checkout master")
    do_command("git svn rebase")
    do_command("git push github master")
  end
end

# How diy was setup on otto:
#
#   cd ~/src
#   git svn clone --stdlayout --prefix=svn/ https://svn.concord.org/svn/diy diy.svn.git
#   cd diy.svn.git
#   git gc
#   git remote add github git@github.com:stepheneb/diy.git
#   git push github master
#
# Then I created a post-commit hook here:
#
#   /home/subversion/diy/hooks/post-commit
#
# added this line:
#
#   /usr/bin/curl -d "revision=$REV&repository=$REPOS" \
#   http://svn.concord.org/cgi-script/post-commit.php
#
# and set the correct permissions:
#
#   sudo chown apache:apache /home/subversion/diy/hooks/post-commit
#   sudo chmod 775 /home/subversion/diy/hooks/post-commit
#
def update_diy
  chdir("/home/sbannasch/src/diy.svn.git") do
    do_command("git svn fetch")
    do_command("git checkout master")
    do_command("git svn rebase")
    do_command("git push github master")
  end
  # chdir("/home/sbannasch/src") do
  #   do_command("nice -n 15 tar czf /web/rails.dev.concord.org/archive/diy.svn.git.tgz diy.svn.git")
  # end
end

# How sds was setup on otto:
#
#   cd ~/src
#   git svn clone -Ttrunk --prefix=svn/ https://svn.concord.org/svn/sds sds.git
#   cd sds.git
#   git gc
#   git remote add github git@github.com:stepheneb/sds.git
#   git push github master
#
# Then I created a post-commit hook here:
#
#   /home/subversion/sds/hooks/post-commit
#
# added this line:
#
#   /usr/bin/curl -d "revision=$REV&repository=$REPOS" \
#   http://svn.concord.org/cgi-script/post-commit.php
#
# and set the correct permissions:
#
#   sudo chown apache:apache /home/subversion/sds/hooks/post-commit
#   sudo chmod 775 /home/subversion/svn/sds/hooks/post-commit
#
def update_sds
  chdir("/home/sbannasch/src/sds.svn.git") do
    do_command("git svn fetch")
    do_command("git checkout master")
    do_command("git svn rebase")
    do_command("git push github master")
  end
  chdir("/home/sbannasch/src") do
    do_command("nice -n 15 tar czf /web/rails.dev.concord.org/archive/sds.svn.git.tgz sds.svn.git")
  end
end

# How concord-projects-common was setup on otto:
#
#   cd ~/src
#   git svn clone -Tcommon --prefix=svn/ https://svn.concord.org/svn/projects/trunk concord-projects-common.svn.git
#   cd concord-projects-common.svn.git
#   git gc
#   git remote add gitorious git@gitorious.org:otrunk-examples/mainline.git
#   git push gitorious master
#
# Then I created a post-commit hook here:
#
#   /home/subversion/projects/hooks/post-commit
#
# added this line:
#
#   /usr/bin/curl -d "revision=$REV&repository=$REPOS" \
#   http://svn.concord.org/cgi-script/post-commit.php
#
# and set the correct permissions:
#
#   sudo chown apache:apache /home/subversion/projects/hooks/post-commit
#   sudo chmod 775 /home/subversion/svn/projects/hooks/post-commit
#
def update_concord_projects_common
  chdir("/home/sbannasch/src/concord-projects-common.svn.git") do
    do_command("git svn fetch")
    do_command("git checkout master")
    do_command("git svn rebase")
    do_command("git push gitorious master")
  end
  # chdir("/home/sbannasch/src") do
  #   do_command("nice -n 15 tar czf /web/rails.dev.concord.org/archive/concord-projects-common.svn.git.tgz concord-projects-common.svn.git&")
  # end
end

commit = YAML.load(MailParser::Message.new($stdin.read).body)
repository = commit['repository'][/.*\/(.*)/, 1]
revision = commit['revision']

if LOGGING
  File.open("/home/sbannasch/process_commit_log", 'a+') {|f| f.write("\n\n#{repository}: r#{revision}\n#{Time.now}\n\n")}
end

case repository
when "ccsailportal"
  update_ccsailportal
when "projects"
  update_otrunk_examples
  update_concord_projects_common
when "diy"
  update_diy
when "sds"
  update_sds
end
Document generated by Confluence on Jan 27, 2014 16:56